Skip to content

Add DIFFENGINE as canonicalization backend#172

Open
Transurgeon wants to merge 163 commits intomasterfrom
diffengine-extractor
Open

Add DIFFENGINE as canonicalization backend#172
Transurgeon wants to merge 163 commits intomasterfrom
diffengine-extractor

Conversation

@Transurgeon
Copy link
Member

@Transurgeon Transurgeon commented Mar 4, 2026

DIFFENGINE uses differentiation (via sparsediffpy) to directly extract concrete A, b, c matrices instead of building parametric tensors through CoeffExtractor. This provides an alternative canonicalization path that can be selected with canon_backend='DIFFENGINE'.

Key changes:

  • Add DIFFENGINE_CANON_BACKEND constant and early return in get_canon_backend()
  • Branch in ConeMatrixStuffing.apply() to call build_diffengine_cone_program()
  • Add DiffengineConeProgram class extending ParamConeProg
  • Handle DiffengineConeProgram in conic_solver.format_constraints()
  • Add vstack converter for diff_engine
  • Update sparsediffpy dependency to >= 0.1.3

Description

Please include a short summary of the change.
Issue link (if applicable):

Type of change

  • New feature (backwards compatible)
  • New feature (breaking API changes)
  • Bug fix
  • Other (Documentation, CI, ...)

Contribution checklist

  • Add our license to new files.
  • Check that your code adheres to our coding style.
  • Write unittests.
  • Run the unittests and check that they’re passing.
  • Run the benchmarks to make sure your change doesn’t introduce a regression.

Transurgeon and others added 30 commits June 16, 2025 23:53
initial attempts at adding a smooth canon for maximum
* adds oracles and bounds class to ipopt interface

* adds some settings and solver lists changes for IPOPT

* adds nlp solver option and can call ipopt

* adds more experiments for integrating ipopt as a solver interface

* passing the problem through the inversion

* add some more extra changes

* adding nlmatrixstuffing

---------

Co-authored-by: William Zijie Zhang <william@gridmatic.com>
Co-authored-by: William Zijie Zhang <william@gridmatic.com>
* adding many tests, new smoothcanon for min, and improvements to ipopt_nlpif

* fixing last two tests

* add another example, qcp

* adding example for acopf

* add control of a car example done

---------

Co-authored-by: William Zijie Zhang <william@gridmatic.com>
* update solution statuses thanks to odow

* removes unusued solver information

---------

Co-authored-by: William Zijie Zhang <william@gridmatic.com>
* getting rocket landing example to work

* add changes to the jacobian computation

---------

Co-authored-by: William Zijie Zhang <william@gridmatic.com>
* adding many more example of non-convex functions

* making lots of progress on understanding good canonicalizations

---------

Co-authored-by: William Zijie Zhang <william@gridmatic.com>
Co-authored-by: William Zijie Zhang <william@gridmatic.com>
Transurgeon and others added 20 commits February 19, 2026 11:54
* Rename ESR/HSR to linearizable_convex/linearizable_concave

Spell out opaque acronyms for clarity per PR review feedback:
is_atom_esr → is_atom_linearizable_convex,
is_atom_hsr → is_atom_linearizable_concave,
is_esr → is_linearizable_convex, is_hsr → is_linearizable_concave,
is_smooth → is_linearizable.

Docstrings clarify that "linearizable convex" means the expression is
convex after linearizing all smooth subexpressions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Adopt three-way atom classification: smooth, nonsmooth-convex, nonsmooth-concave

Replace the two-axis is_atom_linearizable_convex/concave overrides across all
atoms with a single method from the paper's three categories: is_atom_smooth,
is_atom_nonsmooth_convex, or is_atom_nonsmooth_concave. The base class derives
the old linearizable methods for backward compatibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Inline atom-level linearizable checks into expression-level composition rules

Remove the intermediate is_atom_linearizable_convex/concave methods and
_has_dnlp_classification helper, which were only used in the two
expression-level composition rules in atom.py. The three-way classification
(smooth, nonsmooth-convex, nonsmooth-concave) is now used directly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Simplify is_atom_smooth docstring

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Propagate initial values to reduced variables in CvxAttr2Constr

When CvxAttr2Constr creates reduced variables for dim-reducing attributes
(e.g., diag), the original variable's initial value was not being lowered
and assigned to the reduced variable. This caused NLP solvers to fail with
"Variable has no value" during initial point construction.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Handle sparse values when propagating diag variable initials

When a diag variable has its value set as a sparse matrix, extract the
diagonal directly via scipy rather than passing it through np.diag which
does not support sparse inputs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Revert "Handle sparse values when propagating diag variable initials"

This reverts commit e422565.

* Revert "Propagate initial values to reduced variables in CvxAttr2Constr"

This reverts commit d02a758.

* fix pnorm nonsmooth convex

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When CvxAttr2Constr creates reduced variables for dimension-reducing
attributes (e.g. diag=True), the initial value was not propagated to
the reduced variable, causing NLP solvers to fail. This mirrors the
existing value propagation already done for parameters.

Also handle sparse diagonal matrices in lower_value() by using
.diagonal() instead of np.diag() which doesn't accept sparse input.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
#153)

* Make is_linearizable_convex/is_linearizable_concave abstract on Expression

Enforce implementation in all Expression subclasses by uncommenting
@abc.abstractmethod decorators. Add missing implementations to indicator
(convex, not concave) and PartialProblem (delegates to is_convex/is_concave).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* remove is_smooth from max

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Separates test-only utility from production code by moving it to
cvxpy/tests/nlp_tests/derivative_checker.py and updating all 19
test file imports.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Address review comments: remove redundant nonsmooth methods, add types, docs

- Remove is_atom_nonsmooth_convex/is_atom_nonsmooth_concave from base Atom
  class and 10 atom subclasses; use existing is_atom_convex/is_atom_concave
  in DNLP composition rules instead
- Add type annotations and docstrings to NLPsolver, Bounds, and Oracles
  in nlp_solver.py
- Document Variable.sample_bounds with class-level type annotation and
  docstring
- Revert GENERAL_PROJECTION_TOL back to 1e-10 (was loosened for removed
  IPOPT derivative checker)
- Update CLAUDE.md atom classification docs to reflect simplified API

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Move sample_bounds docs from #: comments into Variable class docstring

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…156)

* Extract NLP solving logic from problem.py into nlp_solving_chain.py

Move the ~85-line NLP block from Problem._solve() and the two initial
point methods into a dedicated module. This addresses PR review feedback:
- NLP chain building, initial point logic, and solve orchestration now
  live in cvxpy/reductions/solvers/nlp_solving_chain.py
- Use var.get_bounds() instead of var.bounds so sign attributes (nonneg,
  nonpos) are incorporated into bounds automatically
- Initial point helpers are now private module-level functions instead of
  public Problem methods

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Update NLP initial point tests to use _set_nlp_initial_point

Tests now import and call the module-level helper directly instead of
the removed Problem.set_NLP_initial_point() method.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove debug print and expand comment in best_of loop

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Gate best_of print on verbose flag

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address PR review comments on nlp_solving_chain extraction

- Add circular import comment in problem.py explaining the deferred import
- Move NLP_SOLVER_VARIANTS from nlp_solving_chain.py to defines.py
- Set BOUNDED_VARIABLES = True on NLPsolver base class and use it in
  _build_nlp_chain (matching the conic solver pattern in solving_chain.py)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* adds changes to test_problem with parametrize

* Revert verbose unpack comment to original single-line version

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…160)

* uses new coo handling

* added comment

* fix pre-commit

* fix error message if sample bounds are not set (#161)
* Remove conda from CI, use uv + system IPOPT

Replace conda-based test_nlp_solvers workflow with uv, installing IPOPT
via system packages (apt on Ubuntu, brew on macOS) instead of conda-forge.
Uncomment IPOPT optional dependency in pyproject.toml so uv sync --extra
IPOPT works. Update installation docs in CLAUDE.md and README.md.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Keep IPOPT extra commented out to avoid --all-extras breakage

The test_optional_solvers workflow uses uv sync --all-extras, which would
try to build cyipopt without system IPOPT installed. Instead, install
cyipopt directly via uv pip in test_nlp_solvers where the system library
is available.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove UV_SYSTEM_PYTHON to fix externally-managed env error

uv pip install fails on Ubuntu when UV_SYSTEM_PYTHON=1 because the
system Python is externally managed. Removing it lets uv use its own
venv created by uv sync.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Install LAPACK and BLAS dev libraries for cyipopt build on Ubuntu

cyipopt links against LAPACK and BLAS which are not installed by default
on Ubuntu runners.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Use uv venv + uv pip install instead of uv sync for NLP workflow

uv sync manages a locked environment that doesn't play well with
uv pip install for additional packages. Switch to uv venv + uv pip
install to manage the environment directly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Includes upstream fixes for quad_form canonicalization, gradient handling,
LDL factorization optimization, CI updates, and documentation improvements.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* uses new coo handling

* added comment

* fix pre-commit

* fix error message if sample bounds are not set

* docs for DNLP

* remove whitespace

* split table up into two
* uses new coo handling

* added comment

* fix pre-commit

* fix error message if sample bounds are not set

* docs for DNLP

* remove whitespace

* split table up into two

* cache C problem in best of
- Remove `submodules: recursive` from CI workflows (build, test_nlp_solvers, test_backends)
- Revert `_grad` in affine_atom.py to upstream version
- Remove NaN-allowance block in leaf.py added for NLP structural jacobian
- Update is_dnlp() docstrings to use linearizable terminology
- Add normalize_shape() helper in converters.py, replacing 7 inline occurrences
- Remove unnecessary comment in nlp_solver.py
- Export UNO in cvxpy/__init__.py
- Revert README.md to upstream CVXPY version

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
#168)

* remove unnecessary point in dom + clean up C_problem + clean up oracle

* fix william's comments
…nology (#170)

- Revert whitespace-only changes in huber, abs, ceil, affine_atom, binary_operators
- Revert cosmetic rename and _jacobian_operator addition in index.py
- Revert isnan check in leaf.py back to original error handling
- Restore removed blank line in quad_form.py
- Update DNLP constraint docstrings to use "linearizable convex/concave" terminology

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Remove unused `self.broadcast_type` attribute from `broadcast_to.__init__`
  (was previously used for derivative computation, now handled elsewhere)
- Revert CLAUDE.md to upstream master version (DNLP-specific changes were
  broader than intended for the PR)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@Transurgeon Transurgeon changed the title Add DIFFENGINE as opt-in canonicalization backend Add DIFFENGINE as canonicalization backend Mar 4, 2026
DIFFENGINE uses automatic differentiation (via sparsediffpy) to directly
extract concrete A, b, c matrices instead of building parametric tensors
through CoeffExtractor. This provides an alternative canonicalization path
that can be selected with `canon_backend='DIFFENGINE'`.

Key changes:
- Add DIFFENGINE_CANON_BACKEND constant and early return in get_canon_backend()
- Branch in ConeMatrixStuffing.apply() to call build_diffengine_cone_program()
- Add DiffengineConeProgram class extending ParamConeProg
- Handle DiffengineConeProgram in conic_solver.format_constraints()
- Add vstack converter for diff_engine
- Update sparsediffpy dependency to >= 0.1.3

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@Transurgeon Transurgeon force-pushed the diffengine-extractor branch from 81e1b7b to 67ba4d8 Compare March 4, 2026 15:01
@github-actions
Copy link

github-actions bot commented Mar 4, 2026

Benchmarks that have improved:

   before           after         ratio
 [1cbab94b]       [f3f70f03]
  •     1.20±0s          1.02±0s     0.85  gini_portfolio.Cajas.time_compile_problem
    

Benchmarks that have stayed the same:

   before           after         ratio
 [1cbab94b]       [f3f70f03]
     14.3±0ms         15.1±0ms     1.06  simple_LP_benchmarks.SimpleFullyParametrizedLPBenchmark.time_compile_problem
      493±0ms          520±0ms     1.06  semidefinite_programming.SemidefiniteProgramming.time_compile_problem
     15.1±0ms         15.7±0ms     1.04  simple_QP_benchmarks.ParametrizedQPBenchmark.time_compile_problem
      270±0ms          278±0ms     1.03  slow_pruning_1668_benchmark.SlowPruningBenchmark.time_compile_problem
      11.9±0s          12.1±0s     1.02  finance.CVaRBenchmark.time_compile_problem
      740±0ms          755±0ms     1.02  simple_QP_benchmarks.LeastSquares.time_compile_problem
      1.75±0s          1.78±0s     1.01  simple_QP_benchmarks.UnconstrainedQP.time_compile_problem
      989±0ms          999±0ms     1.01  finance.FactorCovarianceModel.time_compile_problem
      1.40±0s          1.40±0s     1.00  matrix_stuffing.ParamConeMatrixStuffing.time_compile_problem
      238±0ms          238±0ms     1.00  simple_QP_benchmarks.SimpleQPBenchmark.time_compile_problem
      134±0ms          134±0ms     1.00  high_dim_convex_plasticity.ConvexPlasticity.time_compile_problem
      280±0ms          281±0ms     1.00  matrix_stuffing.ParamSmallMatrixStuffing.time_compile_problem
      20.5±0s          20.5±0s     1.00  sdp_segfault_1132_benchmark.SDPSegfault1132Benchmark.time_compile_problem
      311±0ms          312±0ms     1.00  gini_portfolio.Yitzhaki.time_compile_problem
      3.88±0s          3.88±0s     1.00  huber_regression.HuberRegression.time_compile_problem
      875±0ms          873±0ms     1.00  simple_LP_benchmarks.SimpleScalarParametrizedLPBenchmark.time_compile_problem
     40.0±0ms         39.9±0ms     1.00  matrix_stuffing.SmallMatrixStuffing.time_compile_problem
      228±0ms          227±0ms     1.00  gini_portfolio.Murray.time_compile_problem
      10.1±0s          10.0±0s     0.99  simple_LP_benchmarks.SimpleLPBenchmark.time_compile_problem
      679±0ms          670±0ms     0.99  matrix_stuffing.ConeMatrixStuffingBench.time_compile_problem
      4.45±0s          4.38±0s     0.99  svm_l1_regularization.SVMWithL1Regularization.time_compile_problem
      5.14±0s          5.05±0s     0.98  optimal_advertising.OptimalAdvertising.time_compile_problem
      1.60±0s          1.57±0s     0.98  tv_inpainting.TvInpainting.time_compile_problem
      2.75±0s          2.67±0s     0.97  quantum_hilbert_matrix.QuantumHilbertMatrix.time_compile_problem

Transurgeon and others added 5 commits March 4, 2026 11:07
Init derivatives (which allocates dwork scratch space) must happen before
forward evaluation, since quad_form's forward() uses dwork for csr_matvec.
Also reorder constraint init_jacobian before constraint forward for
consistency, and remove redundant problem_init_hessian call.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract the DIFFENGINE special-case from ConeMatrixStuffing.apply() and
conic_solver.format_constraints() into a clean DiffengineMatrixStuffing
sibling class. The solving chain now branches at construction time, and
format_constraints uses polymorphic apply_restruct_mat() dispatch.

- Extract lower_and_order_constraints() for shared constraint lowering
- Add DiffengineMatrixStuffing extending MatrixStuffing (new file)
- Add apply_restruct_mat() to ParamConeProg and DiffengineConeProgram
- Add SymbolicQuadForm converter for QP path through diffengine
- Remove DIFFENGINE branch from ConeMatrixStuffing and isinstance check
  from conic_solver.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants